Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(kinesisfirehose): support Redshift table destination #15837

Closed
wants to merge 294 commits into from
Closed

feat(kinesisfirehose): support Redshift table destination #15837

wants to merge 294 commits into from

Conversation

BenChaimberg
Copy link
Contributor

@BenChaimberg BenChaimberg commented Jul 30, 2021 •
edited
Loading

Kinesis Firehose supports Redshift tables as destinations for delivery streams. This feature provides an L2 construct for this type of destination.

This feature includes the following improvements for users over configuring the destination manually:

  • integrates with aws-redshift module
    • leverages IUser and ITable to allow users to one-click deploy a delivery stream without needing to manually create resources within a Redshift cluster
    • automatically creates user ("assumed" by Firehose to deliver data) and grants minimally necessary permissions on table
  • validates and adds necessary cluster configuration to ensure Firehose can access the cluster
  • automatically creates intermediary resources (S3 bucket, IAM role) needed for Firehose to delivery data to Redshift
  • other generic delivery stream destination features that were created previously (see: readme)

closes #15503

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

madeline-k and others added 30 commits July 19, 2021 13:51
@madeline-k
update README
db9ea21
@madeline-k
Use L1 types in DestinationConfig returned from bind()
49a839e
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
c846f4d 
…deline-k/feat/kinesisfirehose-metrics
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
ff3fb60 
…k/feat/kinesisfirehose-sse
@BenChaimberg
remove DestinationBase, move createLoggingOptions and DestinationProps
a2c85bd 
to dests module
@madeline-k
don't pass role into bind()
569ca01
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
af2cb02 
…deline-k/feat/kinesisfirehose-metrics
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
b392d12 
…k/feat/kinesisfirehose-sse
@madeline-k
remove Node.addDependency's to avoid circular dependency
5ee24d4
@madeline-k
move integ test verification to core module instead of destinations m…
c425f78 
…odule
@madeline-k
update tests and remove addDependency
469bf47
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…
2514430 
…at/kinesisfirehose-compression
@madeline-k
fix build errors from merge
97fb6bc
@madeline-k
udpate implementation after merge
42abd26
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
db31e5a 
…ine-k/feat/kinesisfirehose-prefixes
@madeline-k
update integ expected.json
5389186
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-prefixes' into madeline…
1ae1be4 
…-k/feat/kinesisfirehose-buffering
@madeline-k
update unit tests after merge
1f9c228
@madeline-k
run integ tests and update expected.json
75f1b56
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-buffering' into madelin…
0eb56d3 
…e-k/feat/kinesisfirehose-encryption-dest
@madeline-k
Delete integ.s3-all-properties.expected.json
e3a9ec7
@madeline-k
re-run integ tests and update expected.json
0883ade
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-prefixes' into madeline…
37ece90 
…-k/feat/kinesisfirehose-buffering
@madeline-k
Merge branch 'madeline-k/feat/kinesisfirehose-buffering' into madelin…
867b33e 
…e-k/feat/kinesisfirehose-encryption-dest
@BenChaimberg
feat(kinesisfirehose): destinations can provide resources to be depen…
a307108 
…ded on by the delivery stream (#15676)


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@BenChaimberg
remove unused custom-resources dependency
04d5ae7
@BenChaimberg
rewrite providing IAM role section (include separate destination role…
93cc0e5 
…, update example)
@BenChaimberg
add more IAM role detail
3aa77bd
@BenChaimberg
snapshot spacing
7883929
@BenChaimberg
Merge remote-tracking branch 'origin/master' into madeline-k/feat/kin…
1c65da6 
…esisfirehose-deliverystream
@BenChaimberg
Merge branch 'chaimber/redshift-database-query' of github.com:aws/aws…
20419c8 
…-cdk into chaimber/kinesisfirehose-redshift
@BenChaimberg
fix location of handler name
f533242
@BenChaimberg
Merge branch 'chaimber/redshift-database-query' of github.com:aws/aws…
750a2f0 
…-cdk into chaimber/kinesisfirehose-redshift
@BenChaimberg
refactor in prep of correctly generating table name
2152d53
@BenChaimberg
properly generate table name so update replacements succeed
f5ab16d
@BenChaimberg
Merge branch 'chaimber/redshift-database-query' of github.com:aws/aws…
2aff52e 
…-cdk into chaimber/kinesisfirehose-redshift
@BenChaimberg
do not compare asset hashes in integration test
d9596fb
@BenChaimberg
Merge branch 'master' into chaimber/redshift-database-query
17b9e2f
@BenChaimberg
typo
8add00c
@BenChaimberg
Merge branch 'chaimber/redshift-database-query' of github.com:aws/aws…
7c3a144 
…-cdk into chaimber/kinesisfirehose-redshift
Base automatically changed from chaimber/redshift-database-query to master September 13, 2021 18:40
@BenChaimberg BenChaimberg mentioned this pull request Sep 15, 2021
Closed
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: c4a55d5
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@rix0rrr rix0rrr added feature-request A feature should be added or improved. p1 and removed contribution/core This is a PR that came from AWS. labels Mar 4, 2022
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Mar 4, 2022
@TheRealAmazonKendra TheRealAmazonKendra changed the base branch from v1-main to main June 14, 2022 15:31
@github-actions github-actions bot added the effort/medium Medium work item – several days of effort label Jun 14, 2022
@TheRealAmazonKendra
Copy link
Contributor

This PR has been deemed to be abandoned, and will be automatically closed. Please create a new PR for these changes if you think this decision has been made in error.

@TheRealAmazonKendra TheRealAmazonKendra deleted the chaimber/kinesisfirehose-redshift branch February 14, 2023 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@madeline-k madeline-k madeline-k left review comments

Assignees
No one assigned
Labels
contribution/core This is a PR that came from AWS. effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

(aws-kinesisfirehose): Redshift destination for DeliveryStream
5 participants
@BenChaimberg @aws-cdk-automation @TheRealAmazonKendra @madeline-k @rix0rrr